The Effect of Network Total Order, Broadcast, and Remote-Write Capability on Network-Based Shared Memory Computing1

Emerging system-area networks provide a variety of features that can dramatically reduce network communication overhead. Such features include reduced latency, protected remote memory access, cheap broadcast, and ordering guarantees. In this paper, we evaluate the impact of these features on the implementation of Software Distributed Shared Memory (SDSM), and on the Cashmere system in particular. Cashmere has been implemented on the Compaq Memory Channel network, which supports remote memory writes, inexpensive broadcast, and total ordering of network packets. We evaluate the performance impact of these special network features on the three kinds of SDSM protocol communication: shared data propagation, protocol metadata maintenance, and synchronization, using an 8-node, 32-processor system. Among other things, we compare our base protocol, which leverages all of Memory Channel’s special features, to a protocol based solely on reliable point-to-point messages. We found that the special features improved performance by 18–44% for three of our applications, but less than 12% for our other seven applications. The message-based protocol has the added benefit of allowing shared memory size to grow beyond the addressing limits of the network interface. Moreover, it enables us to implement a home node migration optimization that sometimes more than offsets the advantages of the protocol that fully leverages the Memory Channel features, improving performance by as much as 67%. These results suggest that for systems of modest size, low latency is much more important for SDSM performance than are remote writes, broadcast, or total ordering. At the same time, results on an emulated 32-node system indicate that broadcast based on remote writes of widely-shared data may improve performance by up to 56% for some applications. If hardware broadcast or multicast facilities can be made to scale, they can be beneficial in future system-area networks.